Data processing on memory controller

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing data on a memory controller. One of the methods comprises obtaining a first request and a second request to access respective data corresponding to the first and second requests at a first memory device of the plurality of memory devices; and initiating interleaved processing of the respective data; receiving an indication to stop processing requests to access data at the first memory device and to initiate processing requests to access data at a second memory device, determining that the respective data corresponding to the first and second requests have not yet been fully processed at the time of receiving the indication, and in response, storing, in memory accessible to the memory controller, data corresponding to the requests which have not yet been fully processed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. application Ser. No. 16/930,172, filed onJul. 15, 2020, which claims priority to U.S. Provisional Application No.62/964,031, filed on Jan. 21, 2020. The disclosures of the priorapplications are considered part of and are incorporated by reference inthe disclosure of this application.

BACKGROUND

This specification relates to memory controllers. A memory controllercan be logic circuitry configured to access data by reading from andwriting to memory devices. The memory controller maps data stored inmemory devices and identifies the data by address in the accessedmemory.

Depending on the type of memory device, e.g., DRAM devices, interfacingwith the memory controller, the memory controller is also responsiblefor periodically refreshing and maintaining data in memory, and forenforcing certain data-access constraints for accessing data and that isspecified by the hardware of certain types of memory devices, e.g.,3D-stacked DRAM devices. Certain types of memory devices are alsosubject to data-access constraints that prevent the same device—or insome cases, memory banks on the same device—from being accessed by thememory controller in successive clock cycles. Despite these constraints,3D-stacked DRAM devices are valued for their speed and reduced-powerconsumption over other types of memory devices.

SUMMARY

This specification describes technologies for processing data by amemory controller. A memory controller receives requests to access andprocess data of different sizes from a first memory device of aplurality of memory devices. The controller processes the datacorresponding to the requests by generating chunks of data thatalternate between data corresponding to two or more requests.Periodically, the memory controller receives indications to stopaccessing data from one memory device, and to begin accessing data atanother memory device. When the memory controller receives anindication, the controller determines whether remaining portions of datacorresponding to the current requests being processed have not yet beenfully processed, and in response, “chops off” the remaining portion bystoring the remaining portion or its address in memory internal to thememory controller, e.g., SRAM tables within the memory controller. Bychopping off remaining portions, the memory controller can moreefficiently process data from memory devices, especially memory devicesthat prohibit accessing a same device or same bank of memory on a devicein successive cycles.

Previously, memory controllers finished processing the remainingportions of data for requests to a first memory device, before beginningto access a second memory device. One problem that occurs with thisapproach is that if remaining portions of the requests are of differentsizes, then eventually the request with the longer remaining portionwill be processed without an accompanying request. Not interleaving themultiple requests from the same memory device can result in “bubbles” ofidle cycles in the processing pipeline of the memory controller. Bubblesof idle cycles can additionally translate to bubbles in the utilizationof a data communication bus between a controller and the differentmemory devices, degrading performance. Similar bubbles occur when acontroller attempts to switch between reading and writing data on thesame memory device.

Instead, the memory controller proceeds to access data from a secondmemory device, until receiving an indication to stop accessing thesecond memory device and to begin accessing the first memory device. Thememory controller determines whether a portion of data accessed from thefirst memory device has been previously “chopped off” and stored inmemory internal to the memory controller. In response, the memorycontroller processes the remaining portion before receiving new requestsfrom the first memory device.

Because the memory controller enforces constraints to ensure that datais accessed evenly across all memory devices, after receiving anindication to stop accessing the first memory device, the memorycontroller cannot receive a new request from the first memory device toaccompany the remaining portion of the longer request. Further, if thememory controller alternated requests between the data for the longerrequest and data for a request from the second memory device, the memorycontroller would create bubbles of cycles for accessing different memorydevices on subsequent cycles. The bubbles are worse for architectures ofmemory devices sharing the same data communications bus, e.g.,3D-stacked memory devices, because different memory devices cannot beaccessed in successive cycles.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof, while processing the requests, receiving, by the memory controller,an indication to stop processing requests to access the first memorydevice, and to initiate processing requests to access a second memorydevice of the plurality of memory devices. The controller, afterreceiving the indication, determines that the first and the secondrequest have not been full processed, and in response, stores, in memoryaccessible to the memory controller, data identifying data correspondingto the requests which has not yet been fully processed. The identifyingdata can be one or more memory addresses identifying portions of thedata corresponding to a request stored in a memory controller that hasnot yet been fully processed. In some implementations, the identifyingdata is the remaining portion of data corresponding to the requests thathas not yet been fully processed. In those implementations, theremaining portion is stored in memory accessible to the memorycontroller.

Other implementations of this aspect include corresponding computersystems, apparatus, and computer programs recorded on one or morecomputer storage devices, each configured to perform the actions of themethods.

After storing the remaining portion of data corresponding to therequests from the first memory device, the memory controller begins toprocess data corresponding to requests from a second memory device. Thememory controller initiates interleaved processing of requests from thesecond memory device, until receiving a second indication to stopaccessing data from the second memory device. Upon receiving the secondindication, the memory controller determines whether currently processedrequests from the second memory device have not yet been fullyprocessed, and in response stores the remaining portions of the requestsin memory. Alternatively, the memory controller stores addressescorresponding to the remaining portions in memory internal to the memorycontroller.

The second indication is an indication to the memory controller to beginaccessing data from another memory device. For example, the other memorydevice can be the first memory device, or a third memory device of theplurality of memory devices. If the memory controller receives theindication to begin accessing data from the first memory device, thememory controller determines whether the first and the second requestshave been fully processed, and if not, resumes interleaved processingbeginning with the previously stored remaining portions.

If the second indication is an indication to the memory controller tobegin a third memory device, the memory controller begins to access datastored in the third memory device. While processing data for therequests from the third memory device, the memory controller has storedthe remaining portions of data corresponding to the requests from thefirst and the second memory devices. Also while processing data from thethird memory device, the memory controller can receive a thirdindication. The third indication is an indication to stop accessing datafrom the third memory device, and to begin accessing data from anothermemory device, e.g., the first memory device, the second memory device,or a not-yet accessed fourth memory device.

In some implementations, rather than chopping off and storing data forboth requests after receiving an indication, the memory controllercontinues to process the remaining portions of the data for the requestsuntil one of the requests is fully processed. Then, the memorycontroller stores the remaining portion—or its corresponding address—ofthe data for the other request. In this way, the memory controllerbalances switching between memory devices while also reducing latency infulfilling individual requests that were otherwise close to fullyprocessed before the memory controller received the indication to switchaccessing memory devices.

In some implementations, instead of receiving an indication to switchmemory devices, the memory controller receives an indication to switchreading/writing modes. Some memory architectures impose a constraintpreventing data from being read from and written to (or vice versa) insuccessive cycles. The switch in mode also generates a bubble of cyclesduring which the memory controller is not processing data. By the sametechniques described in this specification, below, the memory controllercan reduce the number of bubbles caused by switching the read/write modeon the memory controller.

The subject matter described in this specification can be implemented inparticular implementations so as to realize one or more of the followingadvantages. A memory controller can more efficiently process requestsfor data of different sizes on a plurality of memory devices. Ingeneral, any memory controller configured to process mixed-size requestscan benefit from improved bandwidth utilization by implementation of thetechniques of this specification. Memory architectures that prohibitsuccessive accesses of data to the same device and/or memory bank, e.g.,high bandwidth memory, can be accessed more efficiently by reducingbubbles of idle cycles. On average, latency in data retrieval is reducedbecause bandwidth utilization is higher. As a result of higher bandwidthusage, requests can be processed at a higher average rate and spend lesstime idle in queue.

Existing controllers can be augmented with minimal modification toimplement the logic for processing requests as described in thisspecification. These improvements can be realized regardless of thenature of the data of the requests, or the nature of the processingperformed by target devices for the requests. Existing software writtento interface with the memory devices and controller does not need to bemodified to implement these techniques.

The details of one or more implementations of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example memory controller coupled to memory devicesthrough a data communication bus.

FIG. 2 is a flowchart of an example process for processing data on amemory controller.

FIG. 3 is an illustration of data processed across two memory devices bya memory controller.

FIG. 4 is a flowchart of an example process for processing data by amemory controller by delaying the switch to a different memory device.

FIG. 5 is an illustration of data processed across two memory devices bya memory controller when the memory controller delays the switch to adifferent memory device.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 shows an example memory controller 100 coupled to memory devices110 through a data communication bus 120. The controller 100 implementsreordering controller 130, interleaving controller 140, and protocolcontroller 150, each of which being special-purpose logic circuityconfigured to perform respective operations described in detail, below.In addition, the controller 100 is communicatively coupled to a targetdevice 160, which is configured to send requests to access data storedin the memory devices 110 and to receive the output of the memorycontroller 100 after performing processing of data stored in the memorydevices 110. Requests to access data stored in the memory devices 110include read and write requests. Data requested to be accessed by thetarget device 160 can vary in size from request-to-request.

The memory controller 100 can be implemented as special-purpose logiccircuitry on or off a processing unit. In some implementations, thememory controller 100 is implemented as an integrated circuit (a “chip”)that is integrated as part of a processing unit of a computer. Theprocessing unit can be any suitable type of processing unit, e.g., acentral processing unit (CPU), a graphics processing unit (GPU),field-programmable gate array (FGPA) or application-specific integratedcircuit (ASIC), such as a tensor processing unit (TPU).

The memory controller 100 can be integrated as part of a CPU serving asa host processing unit for a hardware accelerator. A hardwareaccelerator can have a host CPU for general processing of data beforethe data is processed by the hardware accelerator. For example, the hostCPU may be responsible for receiving data received as network traffic ina network that includes the host CPU, and for streaming the receiveddata to the hardware accelerator. As described in more detail below, inimplementations in which the host CPU implements the memory controller100, the memory controller performs interleaved processing of datareceived from the memory devices 110, and accesses data in the memorydevices 110 according to the requests interleaved in the interleavedoutput.

In this specification, a “memory device” refers to a storage deviceconfigured to store data which can be read from or written to by thememory controller 100. A memory device is organized as a series ofranks, banks, rows, and columns. A memory device has one or more ranks,and each rank represents a plurality of memory banks. Each memory bankis separated into rows and columns, each representing a physicallocation on the device in which memory can be stored.

In some implementations, the memory controller is implemented on a chipseparate from a chip implementing a processing unit, but nonetheless iscommunicatively coupled through physical connections, e.g., a data busor physical connections like wires, or otherwise communicatively coupledwirelessly to the memory devices 110.

The memory controller 100 can access data stored in a memory device byaddress referencing the location of data to-be-accessed, according to anaddress scheme by bank, row, and column of the location of the targeteddata. For ease of explanation, each memory device is assumed to haveonly one rank, but in some implementations the address scheme alsoreferences locations by rank on a device having more than one rank. Thenumber of banks—and whether each bank can be accessed simultaneously ornot—depends on the type of memory of the device.

The memory devices 110 can be implemented: (i) as separate chipscommunicatively coupled by the data communication bus 120; (ii) as partof a single chip; or (iii) a combination of both. The memory devices 110can be devices of different memory types, e.g., one or more DRAM, SRAM,resistive RAM (ReRAM), magneto-resistive RAM (MRAM), ferro-electric RAM(FRAM), spin torque transfer MRAM (STT-MRAM), phase change memory (PCM),double data rate memory (DDR, e.g., DDR4), including graphics DDR memory(GDDR, e.g., GDDRS), and low-power DDR (LPDDR, e.g., LPDDRS), andconductive bridge RAM (CBRAM). The memory devices 110 can also includenon-volatile memory devices.

In some implementations, each of the memory devices 110 can bephysically connected to one another using a 3D die-stacking technique.The memory devices can be stacked according to any suitable technique,e.g., wafer-to-wafer bonding or through-silicon via (TSV). Some examplesof 3D-stacked DRAM include high bandwidth memory (HBM) and hybrid memorycube (HMC).

3D-stacked memory devices share a same data communication bus forcommunicating with a memory controller, and data accesses to differentdevices are serialized rather than performed in parallel; the memorycontroller 110 takes at least one cycle of computation time to switchaccessing data from one memory device to another memory device. Thememory controller 110, through its reordering controller 130, isconfigured to periodically switch from one memory device to another toensure that data is accessed and processed across all devices, insteadof only to a single device.

The memory devices 110 store data to be processed by the target device160. For example, the data can represent inputs to a machine learningmodel that the target device 160 is configured to process to generate acorresponding output. The target device 160 can be any suitablecomputing device configured to receive data for processing as an outputgenerated by the memory controller 100. In some implementations, thetarget device 160 is a hardware accelerator, e.g., a TPU.

Requests to process data can be initially loaded into the memory devices110 according to a client application on one or more client computers.The requests correspond to segments of data to be processed by thetarget device 160. The request can specify the size of the datacorresponding to the request. Data corresponding to requests are loadedinto the memory devices 110, either using the memory controller 100 orby any other suitable technique.

The reordering controller 130 of the memory controller 100 is configuredto obtain new requests for the memory controller 100 to process. Thereordering controller 130 can use any suitable technique for selectingnew requests to process by the memory controller 100. In general, thereordering controller 130 can store requests received by the targetdevice 160 as a “pool” of requests, implemented for example as a queueor as a map table. The reordering controller 130 can apply any suitabletechnique for determining which requests to select from the pool. Forexample, the reordering controller 130 can apply a first-in-first-out orfirst-in-last-out approach to selecting new requests. In someimplementations, the reordering controller 130 selects new requestsaccording to priority levels for each request. The priority levels canbe user-assigned, assigned by the target device 160, assigned by thereordering controller 130 or other system communicatively coupled to thecontroller 100, or some combination of the preceding. The reorderingcontroller 130 is also responsible for enforcing data-access constraintsand controller constraints corresponding to the memory devices 110.

A controller constraint refers to a constraint the memory devices 110imposes by virtue of its hardware on data accesses, i.e., data reads orwrites, to and from the memory devices 110. Another controllerconstraint is a constraint in which the memory controller 100 cannotaccess different rows of the same bank of a memory device in successivecycles. A data-access constraint is a timing in which the memorycontroller 100 periodically stops accessing data from one memory device,and starts accessing data from another memory device. Data-accessconstraints ensure that all memory devices are periodically accessed bythe memory controller 100, given there are pending requests to thosememory devices.

The interleaving controller 140 of the memory controller 100 isconfigured to perform interleaved processing of the data correspondingto the requests selected by the reordering controller 130. Interleavedprocessing refers to the process performed by the memory controller 100of accessing data in the memory devices 110 by alternating betweeninterleaved requests. The memory controller 100 generates an interleavedoutput of data chunks of a uniform size. For example, the memorycontroller can generate an interleaved output of 32-byte chunks,alternating between data corresponding to the first, second, and thirdrequests, and the memory controller processes the requests according tothe alternating interleaved output. By interleaving data correspondingto two or more requests, the memory controller can service requests moreefficiently among memory devices that are configured to not be read fromor written to in successive cycles by the memory controller. Withoutinterleaving, “bubbles” or idle clock cycles are wasted when a memorycontroller waits one or more cycles before being able to read from orwrite to the same memory device again. In some implementations, memorydevices implement multiple banks of memory, where the memory controlleris prohibited from reading from or writing to the same bank of memory insuccessive clock cycles.

The protocol controller 150 of the memory controller 100 is configuredto interface with the target device 160 to access data on the memorydevices 110 according to the interleaved output processed by thecontroller 100. As the protocol controller 150 accesses data accordingto the requests of the interleaved output, the protocol controller 150can stream the accessed data to the target device 160. While only thesingle target device 160 is shown in FIG. 1, in some implementations theprotocol controller 150 is configured to stream accessed data tomultiple target devices, applying any suitable technique fordistributing the controller output, either synchronously orasynchronously, across each of the multiple devices. In someimplementations, each target device receives the same output.Alternatively, each target device receives a portion of the accesseddata as it is streamed out by the memory controller 100.

In some implementations, the interleaved data is streamed to the targetdevice 160 as the memory controller alternates chunks of data from thememory devices 110. By streaming the data to the target device 160, thetarget device 160 can begin processing the chunks of data earlier andwithout waiting for the memory controller to finish the interleavedoutput corresponding to the received requests.

FIG. 2 is a flowchart of an example process 200 for processing data on amemory controller. For convenience, the process 200 will be described asbeing performed by a memory controller configured to read from and writeto a plurality of memory devices in accordance with this specification.For example, the memory controller 100 of FIG. 1, appropriatelyprogrammed, can perform the process 200.

The memory controller obtains 210 a first request and a second requestto access a first memory device of the plurality of memory devices. Therequests specify data of different sizes, for example ranging from 32bytes to 512 bytes, to be accessed by the memory controller, althoughspecific sizes can vary from implementation-to-implementation.

The memory controller is configured to reduce the number of cycles inaccessing the first memory device by alternating accesses to data storedon different banks of the same memory device.

The memory controller initiates interleaved processing 220 of the firstand the second requests. Before the memory controller receives anindication to stop processing requests, the memory controller isconfigured to continuously replace processed requests with datacorresponding to new requests and accessed from the first memory device.Because the first and second requests need not be the same size, thememory controller will often finish processing the shorter request whileinterleaving the shorter request with a longer request, and begininterleaving the remaining portion of the longer request withcorresponding data of a new request. Depending on the hardwarespecification of the memory devices, the memory controller selects thenew request corresponding to data stored on a memory bank of the firstmemory device different than the longer request still being processed.

While processing the first and the second requests, the memorycontroller receives 230 an indication to stop accessing the first memorydevice and to begin accessing a second memory device. The memorycontroller can maintain a data-access constraint for accessing a memorydevice of the plurality of memory devices for a pre-determined period oftime, before switching to another memory device. The subsequent memorydevice for the controller to access can be any memory device that is notthe first memory device. The controller can select the next memorydevice randomly or by any other suitable method, e.g., sequentially byidentifier corresponding to each of the memory devices.

The memory controller determines 240 that one or both of the first andthe second requests have not been fully processed at the time ofreceiving the indication. Not fully processed means that there is someremaining portion of the data of the first request, the second request,or both requests that has not yet been interleaved by the memorycontroller at the time the memory controller receives the indication.

In response to determining that a remaining portion of datacorresponding to the first or second request remains, the memorycontroller stores 250 data identifying data corresponding to therequests which has not been fully processed. In some implementations,the identifying data is the remaining portion of data corresponding tothe first and second requests which has not been fully processed. Insome implementations, instead of storing the remaining portion, thememory controller stores one or more memory addresses corresponding tothe location of the remaining portion in the first memory device. Ineffect, the memory controller “chops off” the remaining portion of thedata corresponding to the requests that were being processed at the timeof receiving the indication, to later return to when the memorycontroller switches back to the first memory device.

The memory controller begins accessing 260 the second memory device ofthe plurality of memory devices. The controller begins interleavingchunks of data from requests corresponding to data stored on the secondmemory device, until receiving an indication to stop accessing thesecond memory device.

At this point, the indication received from the memory controller tostop accessing the second memory device can also be an indication toeither: (i) begin accessing data at the first memory device, or (ii)begin accessing data at a third memory device of the plurality of memorydevices. Regardless of whether the indication corresponds to (i) or(ii), above, the memory controller responds to the indication bydetermining whether the data for the requests at the time of receivingthe indication have been fully processed. If not, the memory controllerstores data (as the remaining portion not yet fully processed, or asaddresses corresponding to the remaining portions) identifying datacorresponding to the requests.

If the memory controller receives an indication to begin accessing dataat the first memory device, then the memory controller next determineswhether portions of data corresponding to requests to access the firstmemory device were stored in memory. If so, the memory controllerresumes processing by first processing the remaining portions of datapreviously determined to have been stored.

If the memory controller receives an indication to begin accessing dataat a third memory device, then the memory controller begins accessingdata to process the third memory device. The memory controller continuesaccessing the third memory device until receiving an indication, whichwill either be to a previously accessed device, i.e., the first orsecond memory device, or to a fourth device not previously accessed bythe memory controller.

Although the description above assumed that the memory controllerinterleaved data from two requests at a time for each memory device, insome implementations the techniques described in this specification areextended for an arbitrary number of requests corresponding to the samememory device. In those implementations, the memory controllerinterleaves chunks of data in the output that alternates between eachrequest, e.g., data from a first request, a second request, a thirdrequest, back to the first request, and so on.

FIG. 3 is an illustration of data processed across two memory devices bya memory controller. Row 310 and row 320 represent data received by thememory controller corresponding to different memory devices over aperiod of time. Row 330 represents the interleaved output generated bymemory controller using the data illustrated in rows 310 and 320.

Requests 312 and 314 are requests to process data from a first andsecond memory bank, respectively, of a first memory device. Data for therequest 312 is represented by right-diagonal lines in the row 310, whiledata for the request 314 is represented by left-diagonal lines in therow 320. For clarity, requests shown in FIG. 3 begin and end with arespective reference numeral, representing the beginning and end of thedata segment corresponding to the request.

Requests 316 and 318 are requests to access data from a first and secondmemory bank, respectively, of a second memory device. Data for therequest 316 is represented by horizontal lines in the row 310, whiledata for the request is represented by vertical lines in the row 320.

Referring to time t₀ in timeline 340 illustrated in FIG. 3, the memorycontroller initially receives the requests 312 and 314 and beginsinterleaved processing of the corresponding data. From time t₀ to timet₁, the memory controller processes data from the requests 312 and 314,resulting in alternating chunks of data as shown in the segment of row330, up until time t₁. At time t₁, the memory controller receives anindication to stop accessing the first memory device, and to beginaccessing the second memory device. The memory controller determineswhether there is remaining data to be processed from the requests 312and 314 that the controller was processing at time t₁. In this case,there is a remaining portion 350 for both the request 314 and therequest 312. The memory controller stores the remaining portion 350 inmemory 352 accessible to the controller, and proceeds to access andprocess data from the second memory device. Beginning at time t₁ anduntil time t₂, the memory controller processes the data corresponding tothe requests 316 and 318. The dots in the illustration before time t₂represent the passage of time between t₁ and t₂.

At time t₂, the memory controller receives an indication to stopaccessing data from the second memory device, and to begin accessing thefirst memory device. If the memory controller determines that therequests being processed at time t₂ were not fully processed, then thememory controller will store the remaining portion of the requests fromthe second memory device. Upon switching back to the first memorydevice, the memory controller determines whether there is a remainingportion of data left un-processed when the first switch occurred at timet₁. In this example, the memory controller determines that it previouslystored the remaining portion 350 in memory, and beginning from time t₂,processes the remaining portion 350 before receiving new requests fromthe first memory device.

In some implementations, the memory controller defers switching from onememory device to another upon receiving an indication. Instead, thememory controller processes the remaining portions of the currentrequests until one of the requests is fully processed. Then, the memorycontroller stores the remaining portion of the request that was notfully processed before switching to the indicated memory device.

By delaying the switch until one of the requests is fully processed, thememory controller can reduce the process latency for the request that isfully processed since the data for the fully serviced request can besent back to target device. In some implementations it can beadvantageous to delay the switch until only one of the requests is fullyprocessed, instead of waiting for both requests. This is because onerequest may be fully processed long before the second request, thereforethe second request is processed without interleaving a subsequent thirdrequest, resulting in wasted bandwidth.

This issue is worsened when the memory controller switches between morethan two memory devices, or when the request is to process aparticularly large segment of data. In the latter case, processing ofthe large segment of data may be interrupted multiple times by memorydevice switches, further delaying the time to fully process a request.

Switches from one source of data to another source are not limited toswitches between memory devices. In general, the subject matterdescribed above can be implemented to facilitate switches from banks orchannels within a memory device, or between any sources of saved memoryin which the memory controller cannot successively read from or write tothe same source of data in back-to-back cycles.

FIG. 4 is a flowchart of an example process 400 for processing data by amemory controller by delaying the switch to a different memory device.For convenience, the process 400 will be described as being performed bya memory controller configured to read from and write to a plurality ofmemory devices in accordance with this specification. For example, thememory controller 100 of FIG. 1, appropriately programmed, can performthe process 400.

After receiving an indication to stop accessing data from a first memorydevice, the memory controller determines 410 that data for requests thatwere being processed at the time of receiving the indication have notyet been fully processed.

Upon determining that data for the requests have not yet been fullyprocessed, the memory controller optionally determines 420 that aremaining portion of a first request currently being processed isshorter than a remaining portion of a second request also currentlybeing processed, e.g., by tracking the remaining portion of datacorresponding to each request while processing the requests. A shorterremaining portion indicates that the memory controller will finishprocessing the shorter portion before the longer portion.

In some implementations, instead of making the determination, the memorycontroller continues processing until data for one of the requests isfully processed. Making the determination allows the memory controllerto further determine whether it should delay the switch at all, asdescribed in more detail, below. Alternatively or in addition, thecontroller is configured to switch after some fixed delay, e.g., anumber of iterations/cycles or a period of time, or after somedynamically-adjusted delay, e.g., an adjusted delay in response to thememory controller load-balancing requests processed for each memorydevice.

Upon determining that the remaining portion of data corresponding to thefirst request is shorter than the remaining portion of the secondrequest, the memory controller processes 430 the first and secondrequests until fully processing one of the requests. In someimplementations, the memory controller determines whether the differencein size of the remaining portions meets a threshold difference, and onlyproceeds to process the shorter remaining portion if it meets thethreshold. In some implementations, if the remaining portions for datacorresponding to each request does not meet the threshold difference,e.g., because the remaining portions are of the same size, then thememory controller proceeds to process both requests to completion bydelaying the switch in memory device until the longer remaining portionor “chop off” both requests without delaying the switch in memorydevice.

The memory controller stores 440 data identifying data corresponding tothe request that has not yet been fully processed. As described abovewith reference to FIG. 2, the memory controller can store the remainingportion of the data for the second request (in this example, the firstrequest is fully processed first) in memory accessible to thecontroller, or the memory controller can store a memory address of thelocation of the remaining portion in the corresponding memory device.

The memory controller begins accessing 450 data at the second memorydevice. The memory controller processes data as described above withreference to FIG. 2, until receiving an indication to stop accessingdata at the second memory device. In some implementations, theindication received from the memory controller is an indication toswitch from reading to writing data.

FIG. 5 is an illustration of data processed across two memory devices bya memory controller when the memory controller delays the switch to adifferent memory device. Row 510 and row 520 represent data received bythe memory controller corresponding to different memory devices over aperiod of time. Row 530 represents the interleaved output generated bymemory controller using the data illustrated in rows 510 and 520.

Similar to the description of FIG. 3, above, requests 512 and 514 arerequests to access data represented by right-diagonal and left-diagonallines, respectively, of a first memory device. Requests 516 and 518 arerequests to access data of a second memory device and are represented byhorizontal and vertical lines, respectively. For clarity, requests shownin FIG. 5 begin and end with a respective reference numeral,representing the beginning and end of the data corresponding to therequest.

At time t₀ the memory controller begins processing data corresponding tothe requests 512 and 514. At time t₁, the memory controller receives anindication to stop accessing data at the first memory device. The memorycontroller determines that there are remaining portions of the data forrequests 512 and 514 to still process. From time t₁ to time t₂, thememory controller processes the remaining portions of the data forrequests 512 and 514 until reaching the end of one of the remainingportions. At time t₂, the memory controller fully processes data for therequest 514. Then, the memory controller saves a remaining portion 550of data corresponding to the request 512. The controller stores inmemory 552 accessible to the controller the remaining portion 550. Insome implementations, instead of storing the remaining portion 550, thecontroller stores the address for the remaining portion 550.

From time t₂ to time t₃, the memory controller processes data for therequests 516 and 518, by accessing the data from the second memorydevice. The dots in the illustration before time t₃ represent thepassage of time between t₂ and t₃.

At time t₃, the memory controller receives an indication to stopaccessing the second memory device, and to begin accessing data in thefirst memory device. In response, the memory controller delays theswitch from the second memory device and stores the remaining portion ofthe requests that were being processed at time t₂, if any. Then, thememory controller determines whether a remaining portion of data fromthe first memory device was stored. In this case, the memory controllerdetermines that the remaining portion 550 was previously stored, andresumes processing beginning with the remaining portion 550. Since theremaining portion is only of a single request (request 512), the memorycontroller also receives a new request 522 to process alongside theremaining portion 550.

Embodiments of the subject matter and the actions and operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, e.g.,one or more modules of computer program instructions, encoded on acomputer program carrier, for execution by, or to control the operationof, data processing apparatus. The carrier may be a tangiblenon-transitory computer storage medium. Alternatively or in addition,the carrier may be an artificially-generated propagated signal, e.g., amachine-generated electrical, optical, or electromagnetic signal that isgenerated to encode information for transmission to suitable receiverapparatus for execution by a data processing apparatus. The computerstorage medium can be or be part of a machine-readable storage device, amachine-readable storage substrate, a random or serial access memorydevice, or a combination of one or more of them. A computer storagemedium is not a propagated signal.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.Data processing apparatus can include special-purpose logic circuitry,e.g., an FPGA (field programmable gate array), an ASIC(application-specific integrated circuit), or a GPU (graphics processingunit). The apparatus can also include, in addition to hardware, codethat creates an execution environment for computer programs, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages; and it can be deployed in any form, including as astand-alone program, e.g., as an app, or as a module, component, engine,subroutine, or other unit suitable for executing in a computingenvironment, which environment may include one or more computersinterconnected by a data communication network in one or more locations.

A computer program may, but need not, correspond to a file in a filesystem. A computer program can be stored in a portion of a file thatholds other programs or data, e.g., one or more scripts stored in amarkup language document, in a single file dedicated to the program inquestion, or in multiple coordinated files, e.g., files that store oneor more modules, sub-programs, or portions of code.

The processes and logic flows described in this specification can beperformed by one or more computers executing one or more computerprograms to perform operations by operating on input data and generatingoutput. The processes and logic flows can also be performed byspecial-purpose logic circuitry, e.g., an FPGA, an ASIC, or a GPU, or bya combination of special-purpose logic circuitry and one or moreprogrammed computers.

Computers suitable for the execution of a computer program can be basedon general or special-purpose microprocessors or both, or any other kindof central processing unit. Generally, a central processing unit willreceive instructions and data from a read-only memory or a random accessmemory or both. The essential elements of a computer are a centralprocessing unit for executing instructions and one or more memorydevices for storing instructions and data. The central processing unitand the memory can be supplemented by, or incorporated in,special-purpose logic circuitry.

Generally, a computer will also include, or be operatively coupled to,one or more mass storage devices, and be configured to receive data fromor transfer data to the mass storage devices. The mass storage devicescan be, for example, magnetic, magneto-optical, or optical disks, orsolid state drives. However, a computer need not have such devices.Moreover, a computer can be embedded in another device, e.g., a mobiletelephone, a personal digital assistant (PDA), a mobile audio or videoplayer, a game console, a Global Positioning System (GPS) receiver, or aportable storage device, e.g., a universal serial bus (USB) flash drive,to name just a few.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on one or morecomputers having, or configured to communicate with, a display device,e.g., a LCD (liquid crystal display) or organic light-emitting diode(OLED) monitor, a virtual-reality (VR) or augmented-reality (AR)display, for displaying information to the user, and an input device bywhich the user can provide input to the computer, e.g., a keyboard and apointing device, e.g., a mouse, a trackball or touchpad. Other kinds ofdevices can be used to provide for interaction with a user as well; forexample, feedback and responses provided to the user can be any form ofsensory feedback, e.g., visual, auditory, speech or tactile; and inputfrom the user can be received in any form, including acoustic, speech,or tactile input, including touch motion or gestures, or kinetic motionor gestures or orientation motion or gestures. In addition, a computercan interact with a user by sending documents to and receiving documentsfrom a device that is used by the user; for example, by sending webpages to a web browser on a user's device in response to requestsreceived from the web browser, or by interacting with an app running ona user device, e.g., a smartphone or electronic tablet. Also, a computercan interact with a user by sending text messages or other forms ofmessage to a personal device, e.g., a smartphone that is running amessaging application, and receiving responsive messages from the userin return.

This specification uses the term “configured to” in connection withsystems, apparatus, and computer program components. That a system ofone or more computers is configured to perform particular operations oractions means that the system has installed on it software, firmware,hardware, or a combination of them that in operation cause the system toperform the operations or actions. That one or more computer programs isconfigured to perform particular operations or actions means that theone or more programs include instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the operations oractions. That special-purpose logic circuitry is configured to performparticular operations or actions means that the circuitry has electroniclogic that performs the operations or actions.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface, a web browser, or anapp through which a user can interact with an implementation of thesubject matter described in this specification, or any combination ofone or more such back-end, middleware, or front-end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network.

The relationship of client and server arises by virtue of computerprograms running on the respective computers and having a client-serverrelationship to each other. In some embodiments, a server transmitsdata, e.g., an HTML page, to a user device, e.g., for purposes ofdisplaying data to and receiving user input from a user interacting withthe device, which acts as a client. Data generated at the user device,e.g., a result of the user interaction, can be received at the serverfrom the device.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of what isbeing claimed, which is defined by the claims themselves, but rather asdescriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially be claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claim may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited inthe claims in a particular order, this should not be understood asrequiring that such operations be performed in the particular ordershown or in sequential order, or that all illustrated operations beperformed, to achieve desirable results. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system modules and components in the embodimentsdescribed above should not be understood as requiring such separation inall embodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

What is claimed is:
 1. A system comprising: a plurality of memorydevices; a memory controller configured to read from and write to atleast the plurality of memory devices and perform operations comprising:obtaining a first request to access a first data of a first sizecorresponding to the first request and a second request to access asecond data of a second size corresponding to the second request,wherein the first and second data are stored at a first memory device ofthe plurality of memory devices, wherein the first size is longer thanthe second size; initiating interleaved processing of the first andsecond data corresponding to the first and second requests, wherein theinterleaved processing comprises alternating between accessing andprocessing the first and second data corresponding to the first andsecond requests, respectively; receiving a first indication to stopprocessing the first and second requests to access the first and seconddata at the first memory device; in response, determining that the firstand second data corresponding to the first and second requests have notyet been fully processed at the time of receiving the first indication;in response to determining the first and second data corresponding tothe first and second requests have not yet been fully processed,storing, in memory accessible to the memory controller, the respectiveremaining portions of the first and second data corresponding to thefirst and second requests that have not been fully processed; receivinga second indication to resume accessing data corresponding to the firstand second requests; in response, resuming the interleaved processingfor the respective remaining portions of the first and second datacorresponding to the first and second requests until the second data hasbeen fully processed but the first data has not yet been fullyprocessed; and re-initiating the interleaved processing of a remainderof the first data corresponding to the first request and another datacorresponding to a new request at the first memory device.
 2. The systemof claim 1, wherein interleaved processing of the first and second datacorresponding to the first and the second requests comprises generatingan interleaved output comprising chunks of data that alternate betweenthe first data corresponding to the first request and the second datacorresponding to the second request.
 3. The system of claim 2, whereinthe operations further comprise streaming the interleaved output toaccess one or more memory devices storing data corresponding to requestsinterleaved in the interleaved output.
 4. The system of claim 1, whereinthe operations further comprise: tracking the respective remainingportions of the first and second data while processing the first andsecond requests; and determining a difference in size of the respectiveremaining portions of the first and second data according to the firstand second requests.
 5. The system of claim 4, wherein during resumingthe interleaved processing for the respective remaining portions of thefirst and second data, the operations further comprise: determiningwhether the difference in size exceeds a threshold difference; and inresponse to determining the difference in size exceeds the thresholddifference, resuming the interleaved processing for the respectiveremaining portions of the first and second data corresponding to thefirst and second requests until the second data has been fully processedbut the first data has not yet been fully processed.
 6. The system ofclaim 1, wherein after re-initiating the interleaved processing of theremainder of the first data corresponding to the first request and theother data corresponding to the new request at the first memory device,the operations further comprise: receiving a third indication to stopprocessing requests to access data at the first memory device;determining that the respective data corresponding to the first requestand the new request have not yet been fully processed at the time ofreceiving the third indication; and in response, determining theremaining portion of the other data corresponding to the new request issmaller in size than the remaining portion of the rest of the firstdata; continuing the interleaved processing of the rest of the firstdata and the other data until the other data has been fully processedbut the rest of the first data has not yet been fully processed, andstoring, in memory accessible to the memory controller, the remainingportion of the rest of the first data corresponding to the first requestwhich has not yet been processed once the other data has been fullyprocessed.
 7. The system of claim 1, wherein after re-initiating theinterleaved processing of the remainder of the first data correspondingto the first request and the other data corresponding to the new requestat the first memory device, the operations further comprise: receiving athird indication to stop processing requests to access data at the firstmemory device; and in response, continuing the interleaved processinguntil a specified period of time elapses.
 8. A method performed by amemory controller configured to read from and write to a plurality ofmemory devices, the method comprising: obtaining a first request toaccess a first data of a first size corresponding to the first requestand a second request to access a second data of a second sizecorresponding to the second request, wherein the first and second dataare stored at a first memory device of the plurality of memory devices,wherein the first size is longer than the second size; initiatinginterleaved processing of the first and second data corresponding to thefirst and second requests, wherein the interleaved processing comprisesalternating between accessing and processing the first and second datacorresponding to the first and second requests, respectively; receivinga first indication to stop processing the first and second requests toaccess the first and second data at the first memory device; inresponse, determining that the first and second data corresponding tothe first and second requests have not yet been fully processed at thetime of receiving the first indication; in response to determining thefirst and second data corresponding to the first and second requestshave not yet been fully processed, storing, in memory accessible to thememory controller, the respective remaining portions of the first andsecond data corresponding to the first and second requests that have notbeen fully processed; receiving a second indication to resume accessingdata corresponding to the first and second requests; in response,resuming the interleaved processing for the respective remainingportions of the first and second data corresponding to the first andsecond requests until the second data has been fully processed but thefirst data has not yet been fully processed; and re-initiating theinterleaved processing of a remainder of the first data corresponding tothe first request and another data corresponding to a new request at thefirst memory device.
 9. The method of claim 8, wherein interleavedprocessing of the first and second data corresponding to the first andthe second requests comprises generating an interleaved outputcomprising chunks of data that alternate between the first datacorresponding to the first request and the second data corresponding tothe second request.
 10. The method of claim 9, further comprising:streaming the interleaved output to access one or more memory devicesstoring data corresponding to requests interleaved in the interleavedoutput.
 11. The method of claim 9, further comprising: tracking therespective remaining portions of the first and second data whileprocessing the first and second requests; and determining a differencein size of the respective remaining portions of the first and seconddata according to the first and second requests.
 12. The method of claim11, wherein during resuming the interleaved processing for therespective remaining portions of the first and second data, the methodfurther comprises: determining whether the difference in size exceeds athreshold difference; and in response to determining the difference insize exceeds the threshold difference, resuming the interleavedprocessing for the respective remaining portions of the first and seconddata corresponding to the first and second requests until the seconddata has been fully processed but the first data has not yet been fullyprocessed.
 13. The method of claim 9, wherein after re-initiating theinterleaved processing of the remainder of the first data correspondingto the first request and the other data corresponding to the new requestat the first memory device, the method further comprises: receiving athird indication to stop processing requests to access data at the firstmemory device; determining that the respective data corresponding to thefirst request and the new request have not yet been fully processed atthe time of receiving the third indication; and in response, determiningthe remaining portion of the other data corresponding to the new requestis smaller in size than the remaining portion of the rest of the firstdata; continuing the interleaved processing of the rest of the firstdata and the other data until the other data has been fully processedbut the rest of the first data has not yet been fully processed, andstoring, in memory accessible to the memory controller, the remainingportion of the rest of the first data corresponding to the first requestwhich has not yet been processed once the other data has been fullyprocessed.
 14. The method of claim 9, wherein after re-initiating theinterleaved processing of the remainder of the first data correspondingto the first request and the other data corresponding to the new requestat the first memory device, the method further comprises: receiving athird indication to stop processing requests to access data at the firstmemory device; and in response, continuing the interleaved processinguntil a specified period of time elapses.
 15. One or more non-transitorycomputer-readable storage media encoded with instructions that, whenexecuted by a memory controller configured to read from and write to aplurality of memory devices, causes the memory controller to performoperations comprising: obtaining a first request to access a first dataof a first size corresponding to the first request and a second requestto access a second data of a second size corresponding to the secondrequest, wherein the first and second data are stored at a first memorydevice of the plurality of memory devices, wherein the first size islonger than the second size; initiating interleaved processing of thefirst and second data corresponding to the first and second requests,wherein the interleaved processing comprises alternating betweenaccessing and processing the first and second data corresponding to thefirst and second requests, respectively; receiving a first indication tostop processing the first and second requests to access the first andsecond data at the first memory device; in response, determining thatthe first and second data corresponding to the first and second requestshave not yet been fully processed at the time of receiving the firstindication; in response to determining the first and second datacorresponding to the first and second requests have not yet been fullyprocessed, storing, in memory accessible to the memory controller, therespective remaining portions of the first and second data correspondingto the first and second requests that have not been fully processed;receiving a second indication to resume accessing data corresponding tothe first and second requests; in response, resuming the interleavedprocessing for the respective remaining portions of the first and seconddata corresponding to the first and second requests until the seconddata has been fully processed but the first data has not yet been fullyprocessed; and re-initiating the interleaved processing of a remainderof the first data corresponding to the first request and another datacorresponding to a new request at the first memory device.
 16. The oneor more non-transitory computer-readable storage media of claim 15,wherein interleaved processing of the first and second datacorresponding to the first and the second requests comprises generatingan interleaved output comprising chunks of data that alternate betweenthe first data corresponding to the first request and the second datacorresponding to the second request.
 17. The one or more non-transitorycomputer-readable storage media of claim 16, wherein the operationsfurther comprise streaming the interleaved output to access one or morememory devices storing data corresponding to requests interleaved in theinterleaved output.
 18. The one or more non-transitory computer-readablestorage media of claim 15, wherein the operations further comprise:tracking the respective remaining portions of the first and second datawhile processing the first and second requests; and determining adifference in size of the respective remaining portions of the first andsecond data according to the first and second requests.
 19. The one ormore non-transitory computer-readable storage media of claim 18, whereinduring resuming the interleaved processing for the respective remainingportions of the first and second data, the operations further comprise:determining whether the difference in size exceeds a thresholddifference; and in response to determining the difference in sizeexceeds the threshold difference, resuming the interleaved processingfor the respective remaining portions of the first and second datacorresponding to the first and second requests until the second data hasbeen fully processed but the first data has not yet been fullyprocessed.
 20. The one or more non-transitory computer-readable storagemedia of claim 15, wherein after re-initiating the interleavedprocessing of the remainder of the first data corresponding to the firstrequest and the other data corresponding to the new request at the firstmemory device, the operations further comprise: receiving a thirdindication to stop processing requests to access data at the firstmemory device; determining that the respective data corresponding to thefirst request and the new request have not yet been fully processed atthe time of receiving the third indication; and in response, determiningthe remaining portion of the other data corresponding to the new requestis smaller in size than the remaining portion of the rest of the firstdata; continuing the interleaved processing of the rest of the firstdata and the other data until the other data has been fully processedbut the rest of the first data has not yet been fully processed, andstoring, in memory accessible to the memory controller, the remainingportion of the rest of the first data corresponding to the first requestwhich has not yet been processed once the other data has been fullyprocessed.
 21. The one or more non-transitory computer-readable storagemedia of claim 15, wherein after re-initiating the interleavedprocessing of the remainder of the first data corresponding to the firstrequest and the other data corresponding to the new request at the firstmemory device, the operations further comprise: receiving a thirdindication to stop processing requests to access data at the firstmemory device; and in response, continuing the interleaved processinguntil a specified period of time elapses.